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ABSTRACT 

In July 1990, ''Standards, Policies and Procedures for 
the Evaluation of Assessment Instruments Used in the California 
Community Colleges 11 was published, requiring colleges to determine 
whether current testing practices discriminate against members of 
underrepresented groups. The standard for demonstrating 
disproportionate impact comes from the Equal Employment Opportunity 
Commission (EEOC) guidelines. Disproportionate impact exists when any 
assessment instrument yields a selection rate for any race, gender, 
or ethnic group which is less than 80% of the rate for the group with 
the highest rate. If evidence for disproportionate impact exists, the 
school must demonstrate that the placement test is valid and not 
biased. Golden West College (GWC) uses the Combined English Language 
Skills Assessment (CELSA) to place non-native speakers of English 
into English as a Second Language (ESL) or English classes. A study 
was conducted at GWC to determine the impact of CELSA testing on 
historically underrepresented groups. The tested population included 
all students assessed in spring, summer, and fall 1993 (N 1C 2,205). The 
sample was broken down by gender, age, language, and disability 
status. The study found no significant patterns of disproportionate 
impact for gender; however, there was possible impact for age, which 
appears to be related to years out of school, and for disability 
status. Follow-up investigations are recommended to explain results. 
(KP) 
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Disproportionate Impact 

1 

Combined English Language Skills Assessment (CELSA): 
Analysis of Disproportionate Impact 

In July 1990, the California Community College Chancellor's Office published Standards, 
Policies and Procedures for the Evaluation of Assessment Instruments Used in the California 
Community Colleges . Section III of that document presents a set of explicit criteria to be met by 
test publishers and another set to be met by the colleges as test users. Generally, these minimum 
standards charge the test publisher with the responsibility to evaluate the reliability, validity, and 
cultural or linguistic bias, insensitivity and offensiveness of placement instruments. In addition, 
the standards charge local community colleges with the responsibility to evaluate the local uses of 
the instrument. One requirement is to determine the impact of testing on members of protected 
student groups (The descriptors ''under-represented groups" and "target groups" will be used 
interchangably in this report). Specifically, colleges must answer the question, do current local 
testing practices discriminate, whether intentionally or unintentionally, against members of 
protected groups? 

The standard for demonstrating disproportionate impact comes from the Equal 
Employment Opportunity (EEOC) guidelines (1978) concerning disproportionate impact in 
employee selection. Generally, evidence for disproportionate impact exists when any assessment 
instrument yields a selection rate for any race, gender, or ethnic group which is less than 
four-fifths (eighty percent) of the rate for the group with the highest rate. A greater than 
four-fifths rate, however, will not be considered evidence of disproportionate impact. For 
example, if 50 white students and 50 black students take an English placement test, and 70% of 



the white students but only 40% of the black students are placed in English 100, evidence exists 
for disproportionate impact because 40 is less than four-fifths of 70. In such a case the school 
must demonstrate that the English placement test is valid, and therefore the difference in 
placement rates is not reflective of inherent test bias. Specifically, those responsible for the school 
testing program must provide evidence that 70 percent of the whi;e students and only 40 percent 
of the black students were indeed prepared for English 100. 

As mentioned earlier, discrimination against a protected group by a selection process 
characterizes disproportionate impact. What exactly, then, defines a protected group? The 
California Community Colleges Standards states that students must be monitored by gender, age, 
ethnicity, and disability status. Although ethnicity is listed as a target classification, linguistic 
background may be an appropriate substitute for ethnicity as a grouping characteristic for the 
evaluation of ESL tests. Ethnic backgrounds where English is the primary language would 
obviously not be included in an ESL evaluation. 

CELSA. All students at Golden West College are expected to take English and 
mathematics placement tests prior to initial enrollment, unless exempted on the basis of having 
earned an associate degree or higher, or by enrolling in six or fewer units without the intent 
pursuing a degree or certificate. Non-native speakers of English, however, are advised to take tl e 
Combined English Language Skills Assessment (CELSA) (Ilyin, 1992) prior to enroilment. 

The CELSA is a language skills measure, which focuses primarily on grammar in a reading 
context. It contains passages appropriate for beginning, intermediate and advanced level students. 
Each passage involves common situations experienced by students and typically encountered in 
teaching materials. The CELSA is appropriate for testing students in community colleges, 
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universities, and high school ESL or foreign language programs. It is not suitable, however, for 
students in adult immigrant open enrollment pre-literate classes (Ilyin, 1992). At Golden West 
College (GWC), this course is ESL 900 (Ilyin, 1992). As such, at GWC, the CELSA is used in 
such a way that students who do not score sufficiently high to place into ESL 001 (Introduction 
to English Language II) "default" to ESL 900 (Introduction to English Language I). 

Golden West College implemented the CELSA in the Spring of 1993 based on the 
supportive evidence of its validity and the recommendations for implementation that resulted from 
the pilot study (Isonio, 1992). That pilot study yielded the initial placement validity evidence and 
generated cut scores that c jmprise a placement rule for ESL at GWC (Table 1). Those cut scores 
were modified slightly prior to the Fall 1993 term. 

According to current cut scores, students scoring below 24 on the CELSA are referred to 
the open enrollment ESL 900 course, as described above. All students scoring above 23 and 
below 34 are referred to ESL 001. Students who score 56 or above are also given the option to 
take the ESL Placement Writing Sample in addition to the CELSA. A final placement is then 
made to ESL 005, English 010, or English 100. 



Table 1 



Current CELSA cut scores for ESL placement recommendations. 



Recommended Level 



CELSA Score 



ESL 900 
ESL 001 
ESL 002 
ESL 003 
ESL 004 



<= 23 
24 - 33 
34-41 
42-50 
51 - 55 
56-61 * 
62-66 
67 - 75 * 



ESL 005 or Engl 010 
Engl 010 

Engl 010 or Engl 100 



* Additional information provided by the student's ESL Placement Writing Sample. 

One recommendation in the pilot study was the need to study the CELSA and the cut 
scores for possible disproportionate impact. The current project is in direct response to the 
recommendations from the CELSA pilot study and the State Chancellor's Office mandate to 
locally validate assessment instruments. Specifically, this project has the purpose of determining 
the impact of CELSA testing on historically underrepresented groups of students. 



Sample . The tested population included all students who were assessed using the CELSA 
during the Spring, Summer, and Fall terms of 1993 (N=2205). The specific sub-populations were 
determined by each of the ESL levels and further divided into special population groups. The 
special subgroups identified for the analysis were determined by the requirements set forth in the 
Standards, Policies, and Procedures for the Evaluation of Assessment Instruments Used in the 
California Community Colleges (1992). The sample was broken down by gender (N=2205), age 
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(N=1588), language (N=2103), and disability status (N=2054) groups (differences in the totals for 
the sub-samples reflect the numbers of non-responing students). 

Design and procedure . Disproportionate impact was assessed by comparing the 
placement rates of each group against 80% of the rate for the group with the highest placement 
rate for each of the ESL levels. A placement rate which is less than 80% of the rate for the group 
with the highest rate (critical rate) will generally be regarded as evidence of disproportionate 
impact. Placement levels were determined by the cut scores presented in 
Table 1. 

Results 

Gender. Table 2 presents the number and placement rate of male and females that were 
recommended to each level of ESL courses based on the CELSA cut scores developed for GWC. 
The critical placement rate is shown in parenthesis. The referral rate for ESL 001 was 12.8% 
(N=151) for males and 14.9% (N=153) for females. Applying the EEOC 80% rule, the critical 
rate for ESL level 001 is 1 1.9% (80% of the female rate of 14.9%). Since the placement rate for 
males exceeds this value, there is no evidence for disproportionate impact for gender of the 
students for ESL level 1. Similarly, no other ESL level indicates a placement rate below its 
respective critical rate. Therefore, there is no evidence for disproportionate impact involving the 
gender of the students across all ESL course levels. 

Disability. Table 3 presents the numbers and placement rates for each ESL level for 
students with self-reported learning disabilities (N=53, or 2.6% of the sample). For ESL levels 
900, 1, 2, and 3, the placement rates for students with self-reported learning disabilities exceed 
their respective critical placement rates. For ESL levels 4, 5, and English 10, however, placement 
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rates are below the critical rate. The EEOC 80% rule results in critical rates of 8.1%, 9.6%, and 
17.5% for levels 4, 5, and 10 respectively, The placement rate into level 4 for students reporting 
a verified learning disability was only 1.9%, for level 5 it was 7.5%, and for level 10, the rate was 
3.8%. Thus, there is evidence of possible disproportionate impact on rates of placement for 
students with self-reported learning disablities at the upper levels of ESL, and into English 010. 

Age. Table 4 presents the numbers and placement rates into each ESL level by age 
category. Results indicated that students under the age of 20 had placement rates lower than the 
critical rates at ESL levels 900 through 003. The placement rates for students between the ages 
of 20 and 29 were lower than the critical rate for levels 900, 3, 4, 5, and Eng 010. Placement 
rates were significantly low for students 30-39 years old at levels 1, 3, and Eng 010. Students 4u 
and above had significantly lower rates at levels 900, 1, 5, and English 010. 
Primary language . 

The disproportionate impact analysis for primary language is impractical, thus it yields 
inconclusive results. An illustration of the fact i? that with 93% of the students being Vietnamese 
and the percentage for all of the rest of the students being only 7%, a change in one person within 
a given level could satisfy the 80% rule for possible disproportionate impact. 

Discussion 

The results yield no significant patterns of disproportionate impact for gender. In the 
cases of age , however, there is evidence of possible dispuportionate impact. Similarly, students 
reporting a verified learning disability have referral rates to the upper ESL levels that are lower 
than the EEOC 80% standard. 



8 



Disproportionate impact appears to be the norm rather than the exception in the California 
Community Colleges. There is evidence for disproportionate impact in over 60% of the studies 
using EEOC guidelines at eleven community colleges (Matriculation Local Research Options 
Committee, 1992). The task at GWC now is to interpret the evidence for disproportionate impact 
by age and for students reporting learning disabilities. 

Many factors affect students' test scores, such as their background characteristics as well 
as an assortment of other variables. The student background variables in this study include 
ethnicity, gender, learning disability status, and age. Moderator variables include, but are not 
limited to number of English courses taken, years out of school, motivational factors, processing 
of the test questions, and possibly test bias (Isonio, 1992). The problem is to identify which 
variable(s) account for the differential placement rates so that appropriate actions can be taken. 

The different placement rates for age do not appear to be due to an unfair test. Rather, 
the differences in referral rates may partially be explained by relative differences in number of 
years out of school. Number of years out of school has been shown to have a negative correlation 
with student preparedness ratings (Thompson, 1994). That is, the longer a student has been out 
of school, the lower the rating by the instructor to the extent to which the student is prepared for 
the class. For the current population, 74% of students younger than 20 years reported being out 
of school less than two years. Only 1% reported being out of school between five and ten years. 
In contrast, 61% of students 30-49 years reported being out of school more than ten years, and 
78% have been out of school at least five years. With older students reporting a higher number of 
years out of school, it follows that students 30-49 years would receive lower preparedness 
ratings. Since referral cut scores are based on preparedness ratings, and number of years is 



negatively correlated with preparedness ratings, older students get lower ratings and younger 
students get higher ratings. 

Lower referral rates for students with self-reported learning disabilities may be related to 
actual differences in performance because of their disability. Students with learning disabilities 
have more difficulties learning in their native language than do other students. It would follow 
that they would have more difficulty learning a second language, which would be reflected in their 
placement into the lower ESL levels at a higher rate. 

Differences in referral rates related to disability status may also be a function of the 
ineffectiveness of the background questions. All background information, on which these group 
comparisons are entirely based, is of a self-report nature. While this tends to be a fairly reliable 
method, it is possible that some students will provide incorrect information because of 
carelessness or misinterpretation of the question (Isonio, 1992). This may be the case especially 
for the learning disability question. The specific wording of the question is "I have a verified 
learning disability (such as dyslexia)"; students may be unsure about the meaning of some of these 
words. Also, since the learning disability category is multidimensional, even if all responses are 
correct, the interpretation is difficult-some disabilities may be related to test performance 
whereas others might not. 
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Recommendations . 



1. Follow-up investigations are necessary to explain why disproportionate impact was 

found in the cases of age and disability. The possibilities outlined in this report 
should be examined. 

2. The "Learning Disability" category, on the CAPP form, should be clarified. This might 

entail specific instructions at the time of assessment, 

3. It may be the case that some of the students reporting learning disabilities should have 

been tested under special conditions. Perhaps information about services, such as 
special testing accommodations, coordinated with the Disabled Students Office 
should be better disseminated. 

4. The content of the student background information questions from the CAPP form 

should be reviewed for appropriateness for ESL students. Either rewording of the 
questions or possibly an alternate, translated form may be developed. The goal is 
to improve the accuracy of the self-report information. 

5. Although not specifically called for by the Standards, differential validity studies, which 

would show whether a test has lower validity coefficients for protected groups, 
should be conducted for all placement tests currently being used at GWC. 
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